feat: support storing different columns in separate files by SkyFan2002 · Pull Request #20193 · databendlabs/databend

SkyFan2002 · 2026-07-23T02:00:13Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Fuse currently rewrites an entire block for an UPDATE, even when only a subset of columns changes. This PR introduces an opt-in partial UPDATE path that stores updated columns in separate physical column-group files while preserving the semantics of a single logical block.

Partial UPDATE requires both the enable_partial_update table option and session setting. Eligible direct UPDATEs write only the changed fields; operations that update all fields, affect cluster-key dependencies, or otherwise do not meet the eligibility requirements fall back to a full block rewrite.

This PR also:

Adds column-group-aware reads for scans, prewhere, row fetch, mutations, compaction, MERGE, and Fuse metadata table functions.
Associates ordinary Bloom indexes with their owning column groups and performs fail-open pruning across multiple Bloom files.
Preserves compression, change tracking, cluster statistics, and HLL metadata across partial updates.
Extends GC and Vacuum reachability to protect active column-group data and paired Bloom files, while cleaning current-format Bloom files associated with deleted data candidates.
Keeps legacy block metadata readable and allows legacy and column-group blocks to coexist.

Column-oriented segments, partitioned tables, computed columns, Inverted/Ngram/Vector/Spatial indexes, Fuse virtual columns, and REPLACE INTO are outside the supported partial UPDATE scope.

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

# Conflicts: # src/query/storages/common/table_meta/src/meta/v2/segment.rs # src/query/storages/fuse/src/io/write/block_writer.rs # src/query/storages/fuse/src/io/write/stream/block_builder.rs # src/query/storages/fuse/src/operations/changes.rs # src/query/storages/fuse/src/operations/common/processors/transform_serialize_block.rs # src/query/storages/fuse/src/operations/mutation/meta/mutation_meta.rs # src/query/storages/fuse/src/operations/mutation/processors/compact_source.rs # src/query/storages/fuse/src/operations/mutation/processors/mutation_source.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c45150b49f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

SkyFan2002 · 2026-07-29T04:52:53Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07225d85e0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

SkyFan2002 · 2026-07-29T06:16:51Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1db5ef3644

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

SkyFan2002 · 2026-07-29T06:29:45Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 866fe7c072

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

SkyFan2002 · 2026-07-29T06:44:59Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 103ae5b508

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

SkyFan2002 · 2026-07-29T06:56:00Z

@codex review

chatgpt-codex-connector · 2026-07-29T07:17:26Z

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 7c0f6343ad

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

SkyFan2002 added 2 commits July 22, 2026 21:00

feat(storage): add column group metadata to blocks

a9edfff

feat(storage): read blocks from column groups

3b51cc5

github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Jul 23, 2026

SkyFan2002 and others added 26 commits July 24, 2026 00:52

feat(storage): support partial updates

2e9975c

fix(storage): harden partial update metadata handling

9ade1e0

fix(storage): protect partial update physical files

6f1cda0

fix(storage): preserve partial update file invariants

84aee80

fix(storage): close partial update integration gaps

6beff4a

fix(storage): harden partial update index compatibility

62fda6f

fix(storage): close partial update bloom lifecycle gaps

1443f8c

fix(storage): bound partial update read amplification

63cb0be

refactor(storage): pair bloom indexes with column groups

d2d797d

fix(storage): address paired bloom review findings

d9f9b7a

fix(storage): close paired bloom layout gaps

bcca08d

refactor(storage): tighten paired bloom normalization

e12c93b

refactor(storage): remove split bloom ngram path

8338fd8

Merge remote-tracking branch 'upstream/main' into wide

bfa0177

refactor(storage): narrow partial update feature scope

7674c8f

fix(storage): address partial update review findings

5ad77e8

fix(storage): close partial update review gaps

3907694

fix(storage): finish partial update review cleanup

8352d96

refactor(storage): simplify partial update data flow

f7d1680

fix(storage): discard stale partial update HLLs

876c070

fix(storage): filter carried partial update HLLs

e227560

fix(storage): normalize replacement HLL metadata

e6963ae

refactor(storage): share partial field remapping

10beca9

Merge branch 'main' into wide

e9acad0

refactor(storage): simplify vacuum2 bloom cleanup

c45150b

SkyFan2002 marked this pull request as ready for review July 29, 2026 03:53

SkyFan2002 requested review from dantengsky and zhyass July 29, 2026 03:53

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/query/service/src/physical_plans/physical_mutation.rs

Comment thread src/query/storages/fuse/src/io/write/block_writer.rs

Comment thread src/query/storages/fuse/src/io/write/block_writer.rs

docs(storage): document partial update limitations

07225d8

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/query/service/src/physical_plans/physical_mutation.rs Outdated

Comment thread src/query/storages/fuse/src/io/write/block_writer.rs Outdated

docs(storage): define partial update support contract

1db5ef3

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/query/service/src/physical_plans/physical_mutation.rs

docs(storage): clarify partial update preconditions

866fe7c

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/query/storages/fuse/src/io/write/block_writer.rs

docs(storage): clarify replace into scope

103ae5b

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/query/service/src/physical_plans/physical_mutation.rs

Comment thread src/query/storages/fuse/src/operations/replace_into/mutator/replace_into_operation_agg.rs Outdated

docs(query): add partial update review rules

7c0f634

SkyFan2002 requested a review from zhang2014 July 31, 2026 04:32

Uh oh!

Conversation

SkyFan2002 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Type of change

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SkyFan2002 commented Jul 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

SkyFan2002 commented Jul 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

SkyFan2002 commented Jul 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

SkyFan2002 commented Jul 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

SkyFan2002 commented Jul 29, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

SkyFan2002 commented Jul 23, 2026 •

edited

Loading